International Journal for the Scholarship of 

Teaching and Learning 


1 V/W I L 

M SCHOLARSHIP of TEACHING and LEARNING 


Volume 6 | Number 2 


Article 26 


7-2012 


Student Opinions and Preferences Regarding 
Personal Response Systems in the Graduate 
Physical Therapy Classroom: A Mixed-Methods 
Inquiry 


Audi Beth Mincer 

Armstrong Atlantic State University, andibeth.mmcer(® armstrong.edu 


Anne W Thompson 

Anne.Thompson(2> arms trong.edu 


Recommended Citation 

Mincer, Andi Beth and Thompson, Anne W. (2012) "Student Opinions and Preferences Regarding Personal Response Systems in the 
Graduate Physical Therapy Classroom: A Mixed-Methods Inquiry," International Journal for the Scholarship of Teaching and Learning: 
Vol. 6: No. 2, Article 26. 

Available at: https://doi.org/10.20429/ijsotl.2012.060226 





Student Opinions and Preferences Regarding Personal Response Systems 
in the Graduate Physical Therapy Classroom: A Mixed-Methods Inquiry 


Abstract 

Little investigation has been conducted on the use of Personal Response Systems (PRS) in either graduate- 
level courses or health professions education. Through anonymous participation in focus groups, graduate 
physical therapy students described specific aspects of PRS that they felt facilitated their learning, as well as 
aspects that hindered their learning. A Likert-type survey was constructed based on focus group outcomes 
and was offered to the entire population of physical therapy students at our institution. Results indicated that 
PRS was perceived to be useful for examination preparation, application of concepts, facilitation of discussion, 
and immediate feedback. Participants perceived cost and technical issues, including lack of faculty technical 
expertise, as problematic. Students exhibited a strong preference for ungraded in-class quizzes, followed by 
provision of these quizzes to students for later study. This unique mixed-method design maximized the use of 
online technology for obtaining both qualitative and quantitative outcomes. 
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Abstract 

Little investigation has been conducted on the use of Personal Response Systems (PRS) in 
either graduate-level courses or health professions education. Through anonymous 
participation in focus groups, graduate physical therapy students described specific aspects 
of PRS that they felt facilitated their learning, as well as aspects that hindered their 
learning. A Likert-type survey was constructed based on focus group outcomes and was 
offered to the entire population of physical therapy students at our institution. Results 
indicated that PRS was perceived to be useful for examination preparation, application of 
concepts, facilitation of discussion, and immediate feedback. Participants perceived cost and 
technical issues, including lack of faculty technical expertise, as problematic. Students 
exhibited a strong preference for ungraded in-class quizzes, followed by provision of these 
quizzes to students for later study. This unique mixed-method design maximized the use of 
online technology for obtaining both qualitative and quantitative outcomes. 

Keywords: Personal response systems; graduate education; health professions education; 
scholarship of teaching and learning (SoTL); student perceptions, higher education 


Introduction 

Personal Response Systems (PRS), also known informally as "clickers," consist of small 
handsets used to send wireless signals from each student to a computer used to display 
results on an LCD projector. Responses are tallied by the computer and displayed on the 
same projector screen in a variety of formats. Quizzes most often consist of multiple-choice 
questions; these questions, along with answer options, can be constructed on any topic, at 
the discretion of the instructor. Instructors using PRS can track and record individual 
student responses and participation, and attendance can also be recorded through 
participation in PRS quizzes. Quizzes may be constructed as stand-alone activities or 
inserted into slideshow presentations. 

The use of a Personal Response System began in our physical therapy curriculum three 
years prior to this study. Faculty members had no prior experience using this technology, 
but had become interested through informal communication with faculty members at a 
different institution. Subsequently, PRS was incorporated into several different types of 
courses including anatomy and kinesiology, basic and advanced clinical concepts, and 
clinical problem-solving. The primary investigator used PRS in most of her courses and 


https://doi.org/l0.20429/ijsotl.2012.060226 


1 


Student Opinions and Preferences Regarding Personal Respoi 


received frequent student comments related to their undergraduate experience with PRS, as 
well as comments about PRS quizzes as they were currently being used in our curriculum. 
Students appeared to have valuable insights about the types of PRS questions and activities 
that they felt facilitated the most effective learning. This research project was 
designed to explore and document their opinions and preferences regarding PRS, but also to 
assess whether these opinions and preferences accurately represented those of the students 
as a whole. Student perceptions and preferences provide a valuable source of information 
that can help faculty members construct their PRS activities to maximize student 
engagement, satisfaction, and learning. 

Student participation during PRS quizzes has been widely reported to be higher than during 
other forms of quizzing used during lectures, such as hand-raising (Corcos & Monty, 2008; 
Gauci, Dantas, Williams, & Kemm, 2009; Mayer et al., 2009; Shaffer & Collura, 2009; 
Stowell, Oldham, & Bennett, 2010; Weerts, Miller, & Altice, 2009). It has also been 
observed that females are less likely to participate using hand-raising as compared to their 
participation using PRS (Graham, Tripp, Seawright, & Joeckel, 2007). Also, student 
responses have generally been found to be more honest when using PRS than when using 
other forms of student response to questions during lectures, such as response cards. 
Students using response cards have been observed looking around at other student 
responses before committing to their own, which often seemed to follow the majority of 
responses made by other students. 

A recent review of PRS in higher education reported that the vast majority of published 
research on the use of PRS in higher education is related to use in large introductory 
physics, biology, and psychology courses (Caldwell, 2007). Two of the most well-known 
scholars in the use of PRS are Eric Mazur (2001) in physics, and Bruff (2009) in several 
disciplines, primarily mathematics. More recent investigationsinclude other large lower- 
division courses such as chemistry (Donovan, 2008) and sociology (Mollborn & Hoekstra, 
2010). It appears that impact assessment of PRS in smaller and/or upper-division courses 
has just begun (Reyerson, Mummey, & Higdon, 2011; Sevian & Robinson, 2011, 2011; 
Smith, Trujillo, & Su, 2011). The overall number of scholarly PRS publications has increased 
dramaticially in the past few years (an average of 10 ERIC higher education citations per 
year 2005-2008; 21 per year 2009-2011), but research in some areas of education is still 
lacking. Very little has been published on the use of PRS in graduate-level courses (Sevian & 
Robinson, 2011). Of the only 21 publications (an average of 3.5 per year) related to health 
professions education and the use of PRS, 17 involve the undergraduate nursing classroom, 
and none are in physical therapy or any other rehabilitation science. 

The current investigation used a novel mixed method approach, which benefitted from both 
qualitative and quantitative aspects. The focus group is a standard qualitative method that, 
while including some facilitation for addressing general areas of inquiry, also encourages 
open participation and interaction. This open approach allows participants to express their 
opinions more openly than if they had been asked to respond to a survey constructed by 
someone outside of their experience, who may only be guessing at what areas respondents 
might want to address. The statements made by participants are more likely to reflect the 
actual opinions of the group that they represent, and the second phase of this particular 
methodology allows verification of the responses. Statements made in the focus group were 
converted into survey items, and the resulting survey was administered to the entire 
population of interest. Survey responses revealed whether statements made by the focus 
group actually reflect those of the larger group. Another unique aspect of the methodology 
used in this particular investigation was that the focus group was conducted in the online 
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environment, which allowed focus group members to participate anonymously, and also 
allowed students in remote locations to have an equal opportunity to participate. 


Higher Education Student Perceptions of PRS 

One of the difficulties in reviewing this literature was the lack of a universally accepted term 
for this technology. The terms "Personal Response System" and "Clickers" are frequently 
used; but there are others, such as "Student Response System," "Electronic Response 
System," and "Zappers." Many publications describe faculty members' experiences with and 
opinions of this technology. Increasing numbers are beginning to describe the results of 
systematic inquiry or assessment of student opinions and perceptions. 

This technology has been in use since the early nineteen-seventies (Mackenzie, 1970; 
Bessler & Nisbet, 1971; Sandler & Bowles, 1974), although the systems have evolved 
dramatically since that time. Faculty members teaching in undergraduate sciences, 
especially physics, have written most of the PRS publications. Very few health professions 
are represented, and nearly all of the published PRS research involved applications in large 
lecture classes in which PRS was used to add an active learning component to traditional 
lectures. Student feedback has been sought almost exclusively through instructor-created 
questionnaires. While these questionnaires offer respondents the opportunity to add written 
comments, there has been minimal qualitative inquiry to assess other concerns that may 
not have been previously identified by instructors and incorporated into the instruments 
that they created. 

In a comprehensive review article, Caldwell (2007) reviewed PRS use in large higher 
education classrooms. She found that students in these studies generally approved of PRS 
use, and that they especially liked the anonymity and immediate feedback that this 
technology provided. Students also reported that they appreciated being able to compare 
their answer to others and collaborate via discussions, and that use of PRS reinforced their 
learning. Caldwell recommended that faculty members "Shut up and listen to students to 
find out how they think, and pay particular attention to wrong answers" (p. 18) if they are 
interested in maximizing student learning from PRS activities in their classrooms. 

Donovan (2008) described student responses to the use of PRS to administer what he 
referred to as ConcepTests in a chemistry class. Eric Mazur, a well-known advocate of peer 
instruction, including PRS, developed ConcepTests in physics (Mazur, 1997). In the Donovan 
study, students were shown a question to which they responded individually using their PRS 
handset. Peer discussion followed viewing of the responses by the entire class, and then 
students were asked to re-vote on the same question. Students felt that the process of 
voting, followed by peer instruction and then re-voting, increased their understanding. This 
positive assessment of the effectiveness of ConcepTests combined with PRS was validated 
through correlation of scores on test items related to those previously addressed in the 
ConcepTests. Participants were asked what they liked about PRS; they did say they enjoyed 
using the technology but even more of the positive comments were related to how it helped 
them achieve mastery over the course material. Donovan noted that a key portion of the 
implementation of PRS in the classroom was the discussion of the correct answer by either 
the instructor or a student. Even if the vast majority of the students answered correctly, 
those who did not get the correct answer need to be exposed to the correct answer and the 
reasoning process used to obtain it. 
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Most students have reported that PRS increases their engagement in classroom activities 
and that this facilitates their learning (Carnasciali, 2009; Hatch, Jensen, & Moore, 2005; 
Nelson, 2008; Shaffer & Collura, 2009). In one of the few investigations that included a 
qualitative component, in this case face-to-face interviews that followed participation in 
questionnaires, Gauci (2009) found that physiology students wanted more PRS questions 
based on concepts rather than facts, and more think-pair-share PRS activities. The most 
common comment made by these students during the interviews was that PRS "made me 
think" (p. 66). These students also preferred that PRS quizzes be non-graded. This 
preference was expressed by the majority of 688 undergraduate students in a separate 
investigation of PRS use in seven different content areas (Graham, et al., 2007). 

Other positive student perceptions cited are that PRS quizzes increased the ability to focus, 
"broke up" lectures, and "lightens up the classroom experience" (Nelson, 2008). The same 
students also reported that they benefited from instant feedback and the ability for 
instructors to use feedback to elaborate on important concepts. Discussion was cited as an 
important aspect of what introductory psychology students felt made clickers successful, 
and these students felt it was important for faculty members to allow adequate time for this 
(Shaffer & Collura, 2009). Participants in the large investigation (7 departments and 688 
students) conducted by Graham (2007) specifically felt that PRS was better used for 
empowerment rather than enforcement. They liked having the opportunity to participate, 
but resented being obligated to participate when attendance or their grade depended on it. 
Carnasciali, et al. (2009) conducted a large investigation of freshmen (427 students) in a 
variety of core classes. He reported that males enjoyed the discussion used in conjunction 
with PRS quizzes significantly more than females, and also that students with a 
Concrete/Reflective learning style liked learning from peers significantly more than students 
with other learning styles. Perceptions of upper division nursing students are also generally 
positive, and the students consistently report appreciating the instant feedback and the 
usefulness of the technology with test preparation (DeBourgh, 2008; Meedzan & Fisher, 
2009; Revell & McCurry, 2010). 

There are also negative aspects attributed to PRS. Cost and technical problems emerged as 
the most common student complaints (Caldwell, 2007; Donovan, 2008; Gauci et al., 2009; 
Graham et al., 2007; Hatch et al., 2005). Technical problems were seen as particularly 
troublesome when PRS activities contributed to course grades (Graham et al., 2007). The 
few negatives reported by students in Caldwell's review (2007) included cost, problems with 
the technology, and the amount of class time used. Students also disliked that the systems 
were used to monitor attendance, which they interpreted as forced attendance (Graham et 
al. 2007). Using technology just for technology's sake and anxiety over grading were viewed 
negatively by students, as were setup and breakdown time (Hatch et al., 2005). Also, 
students sometimes felt that PRS activities wasted class time (Gauci et al., 2009). 


Methods 

The present investigation was conducted simultaneously with three cohorts of students in a 
Doctor of Physical Therapy degree program. One group of students was in the first year of 
the three-year professional program (N = 18), one in the second year (N = 20), and one in 
the third year (N = 22). In the first phase of the study, focus groups were conducted 
separately with students selected from each of the three cohorts. Each total cohort was 
stratified into high, medium, or low academic performance groups based on overall grades. 
All students in this curriculum are required to achieve at least 80% on each major 
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examination and each overall course grade, so all stratification occurred between the 80- 
100% levels. This stratification was conducted and agreed upon by consensus of the 
investigators. After stratification, two names were randomly selected from each group. In 
this way, a focus group was created for each cohort made up of two members from each of 
the high, medium and low achievement strata. Students were invited to participate by e- 
mail from the primary investigator, who uses PRS regularly. Potential participants were told 
that a different faculty member (the secondary investigator) would facilitate the focus group 
and that their participation would be anonymous. If a selected student declined to 
participate, the selection process was repeated from the same stratum until a six-member 
focus group was formed for each cohort. 

The two investigators created a list of guiding questions (see Appendix) based on the 
experiences of the primary investigator using PRS in physical therapy classes, and on 
informal observations and student comments made during these classes. Focus groups were 
conducted online using synchronous chat within our course management system. Three chat 
rooms were created, one for each cohort's focus group. The second investigator, who did 
not use PRS, facilitated the focus groups and was not involved in participant selection. Each 
student who agreed to participate was given electronic permission to enter the chat room by 
the facilitator. When students logged in, they created an alias that was not revealed to 
either the primary investigator or the focus group facilitator. The facilitator and participants 
could only identify each other through their self-selected aliases. Participants were asked to 
allot one to two hours for each of these chats, but the facilitator allowed each group to 
converse as long as desired. All participants knew that the focus group topic was their 
perceptions of PRS, and they were given open opportunities to voice their opinions. The 
facilitator used the guiding questions if groups did not discuss these areas spontaneously. 

Each focus group lasted 60-70 minutes. Technical difficulties prevented entry into the chat 
room by some participants, and others who had agreed to participate did not attempt to log 
in. Since the identities of all participants were blinded, the investigators were unable to 
determine who had or had not participated in each focus group. The two investigators 
analyzed printed transcripts individually to identify common themes and comments, then 
consolidated the lists into a mutually acceptable list. The resulting list of themes and 
comments was converted into a list of statements, which was entered into a five-option 
Likert-type online survey 

( http://www.pt.armstronq.edu/mincer/SurvevPTstudentclickeropinions.pdf ). 

All students from the three cohorts (N = 60) were invited by e-mail to respond to the survey 
anonymously. Each respondent had the opportunity to add comments to each question and 
at the end of the survey. Reminders were sent to all students while the survey was open. 

The Institutional Review Board of Armstrong Atlantic State University approved this 
investigation, and student participation was voluntary and anonymous. 

In essence, this methodology allowed focus group participants to create the survey to which 
each member of the population (including focus group participants) had the opportunity to 
respond. General agreement or lack of agreement between the survey respondents and the 
focus group participants were thus revealed. Survey results provided the researchers with a 
more accurate picture of the perceptions of the entire population, as opposed to only those 
of the small subset that participated in the focus groups. 


Results 


https://doi.org/l0.20429/ijsotl.2012.060226 


5 



Student Opinions and Preferences Regarding Personal Respoi 


Numbers of focus group participants and survey respondents can be seen in Figure 1. An 
overall response rate of 72% was obtained on the survey. Response rates were similar for 
each class: 72%, 80%, and 64% for first, second, and third year students, respectively. 

First year students comprised 30% of the total number of respondents, while second and 
third year students comprised 37% and 33%, respectively, of the total. 

The themes identified in the focus groups, in order of numbers of comments from most to 
least, were: effectiveness, technological problems, question types, graded versus ungraded 
quizzes, and cost. The following chat transcript excerpts briefly illustrate each of these 
themes. Quotes are reproduced exactly as each student typed them in during the chat 
room, including informal abbreviations, misspellings and other errors, because these more 
authentically capture the 'voice' of each participant. 

Effectiveness: Students were overwhelmingly positive about how PRS or "clicker" quizzes 
helped them learn and made many comments about which aspects they thought were most 
effective. One wrote that he or she thought PRS was beneficial, "...because it really makes 
you think it through alone and see if you know an answer. Often times when a question is 
just asked to the whole class someone elses answer might make sense when they say it but 
I may not know if I would have been able to come up with it on my own. I think i learn the 
material well after I have to think on my own first and then it's followed up with a 
discussion." Another student also appreciated the usefulness of each student being asked to 
answer questions in advance of revealing the correct answer and the rationale: "helps 
solidify the info when you have to come up with an answer on your own before the 
discussion begins." One added that because all students were asked to answer each 
question, "It gives the quiet people a chance to answer questions which are normally 
answered by the same crowd of people." Many commented about how important they felt 
the follow-up explanations and discussions were: "i like how they are used now to facilitate 
discuss and make us think," and, "the discussions were almost always helpful, and clarified 
things for most everyone." Several participants felt that they achieved deeper and higher 
quality learning through PRS quizzes: "I think they help to teach logical thinking," "when we 
are done with the clicker quiz, I feel like I actually understand the logic behind the answers 
so that helps me answer questions that are about the same type of logic," "the practice 
questions give a chance to think critically about subject matter," and, "I think we're gaining 
useful deductive reasoning skills with the discussion quizzes." Several appreciated that 
quizzes gave them an opportunity to apply information, including case-based questions: "I 
like them for practice questions because it helps us to start applying some of the 
information," and, "good way to go over many topics/cases." The clickers also "allowed an 
anonymous way to assess your own knowledge of the subject matter;" and while online 
quizzes through course management software also provide the ability to self-assess, "there 
wasn't the instant feedback with the [online] quizzes" to which another student added "or if 
you were confused." This study was not designed to make direct comparisons to online 
quizzes, but nearly all participants preferred in-class PRS with explanation and discussion to 
online quizzes. 


Technological: Many comments were related to the technical problems participants had 
observed during their undergraduate courses, but several felt that these problems were 
related to the less advanced equipment that was being used at that time. The comments 
about their current experience with PRS mostly related to more random, small scale (but 
still important) technical problems, such as device pairing issues, ("sometimes it doesn't 
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work and you can't connect,") intermittent system setup problems, or instructors that 
appeared to have difficulty operating or troubleshooting the system. The last point was 
illustrated when a student wrote that the technical problems were "not normally a 
malfunction of the clickers," and clarified that he or she was referring to user error on the 
part of the professor. The other comments concerning technical problems clustered into two 
themes: the fairness of grading students on performance or attendance when technical 
malfunctions cloud the results, and the loss of valuable classroom time due to technical 
problems. One of the participants gave the following advice to instructors: "make sure you 
know how to use the system to avoid errors costing quality classroom time." 

Problems: Students did not spontaneously express many negative aspects of PRS but were 
asked to describe the biggest disadvantages. The bulk of these replies related to cost, which 
is discussed as a separate theme below. The next largest area of concern was the amount of 
class time that quizzes consumed: "some 'quizes' would end up taking half the class time" 
(in this curriculum, this means a quiz took more than an hour). When asked whether they 
thought the quizzes were a good use of class time, respondents agreed that it depended on 
the nature of the quiz. For example, one responded wrote that it "depends on if the material 
is discussed afterwards or not." This echoes the large number of responses related to the 
perceived effectiveness of the follow-up explanations and discussions and that the benefit of 
the discussion justified the class time invested. One felt that excessively lengthy quizzes 
"become to tedious," and another recommended that instructors should "try not to have the 
whole session take more than 30-45 mins" (class sessions in this curriculum last 2-4 hours 
each). Another concern reported by a couple of respondents was the anxiety that an in-class 
quiz can provoke: "I don't particulary like them as quizzes in class because if you are one of 
the last couple to answer people start rushing you and it tends to be a distraction." This 
concern emerged as a separate issue from the anxiety produced if the quizzes were graded, 
and was more related to being one of the slowest respondents in class often requiring 
everyone else to wait for them to finish. One final concern related to the LCD projector as 
the medium for display: "I don't think they are good for anatomy pictures on the projector 
because of the picture quality." Another respondent said the same thing about questions 
based on imaging studies (such as radiographs, for example) that were embedded in the 
quiz questions; the images were not clear enough to interpret accurately. 

Question types: The students distinguished between types of questions used with PRS. One 
wrote that he or she liked "questions that were more clinically applicable . . . that presented 
some sort of case study, and asked for use of clinical reasoning to determine the correct 
answer, rather than questions that just asked for simple definitions or simple answers along 
those lines." Another wrote that he or she would like professors to write "quality questions 
that cover the material the you feel is most important and will facilitate excellent discussion 
for students." Again, the discussion phase emerged as important to students. One of the 
benefits of PRS is in preparing students for examinations. The participants stated that PRS 
questions were "more helpful when they were the same calibur as test questions." The 
structure of the questions also mattered for their performance in class: "long answer 
choices would take longer to read," and, "questions w/ LONG answers as choices are 
sometimes confusing." As with the earlier report of the lack of clarity in projected 
radiographic images, the lengthy response options were reported as problematic. Lengthy 
question stems and/or response options may not fit on the screen and be viewed in their 
entirety, limiting the students' ability to see the problem set as a whole. Graded versus 
ungraded: These students overwhelmingly preferred that PRS be used without grades 
attached. This seemed to relate more to the anxiety that grading provoked and that concern 
over grades shifted the emphasis away from real learning: "I like the use of clickers as a 
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teaching device, but no so much for graded quizzes or tests." "I like using them as a non- 
graded exercise that then prompts discussion." "If there is to much emphasis on grades it 
detracts away from the point of a quick self-assessment." "I think in the beginning when ppl 
thought they would be graded more people were too worried about their grade and less 
about understanding concepts." Another participant wrote that graded quizzes were "less 
beneficial" because "they became stressful and less of a learning environment." "I learn 
better when I'm not under pressure to make a certain grade." 

Cost: The students were required to buy the PRS handsets and then pay a registration fee 
for each semester of use. While the students viewed PRS as an effective learning tool, they 
became frustrated when required to register for a semester in which the instructor rarely 
used the system: "if I'm paying for them i want to use them, that was one of my biggest 
complaints." Another wrote, "if you are going to use them, use them...don't have us pay $20 
to activate them and use them 3 times." "The biggest disadvantage is the cost." Students 
didn't specify how much use would justify the cost, but did suggest that a frequency of 
perhaps once a week would be a justifiable balance of class time necessary for PRS use. 

The survey consisted of 34 questions constructed from these focus group transcripts. Some 
of the questions related to the students' varied undergraduate experiences, but the majority 
related to their perceptions of PRS as used in the graduate physical therapy curriculum. The 
results described here are their perceptions and preferences related to the use of PRS 
during their graduate physical therapy instruction. "Agree" and "Strongly Agree" responses 
were collapsed into a single response category, which was compared to a collapsed 
"Disagree" and "Strongly Disagree" category. 

The survey responses paralleled the themes and opinions expressed in the focus groups. On 
all items, most survey respondents agreed with the statements distilled from the focus 
group transcripts; but on the items in Figure 2, respondents expressed particularly strong 
agreement. 



Figure 1 . Graphic representation of study procedures 
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These are the items in which at least 80% of respondents agreed, in order of most to least 
agreement. The 'disagree' responses are included to demonstrate that not only did the vast 
majority of students agree, but also that an extremely small proportion of students 
disagreed with any of these statements. 


Summary of Survey Statement 

Agree % 

Disagree % 

Like havinq the Qs & As available after the quiz for review 

100 

0 

Frustratinq to buy and only use 2-3 times 

95 

0 

Good way to practice applyinq information 

95 

5 

Useful for facilitatinq discussion & explanation 

93 

2 

Help me with exam prep if questions are similar 

93 

0 

Good way to qet immediate feedback 

91 

<1 

Useful for reinforcinq important material 

91 

< 1 

Good way for instructor to assess a qroup's overall understandinq 

91 

<1 

Good way to self-assess 

88 

5 

Good way to see what an instructor thinks is important 

88 

5 

Can promote development of critical thinkinq and problem solvinq 

86 

5 

Usually a qood use of class time 

84 

<1 

Like because I can qet feedback without beinq embarrassed 

84 

5 

I prefer unqraded quizzes to qraded ones 

81 

0 


Figure 2. 'Agree' and 'Strongly Agree' responses were collapsed into a single 'Agree' category. The 
same procedure was used to create a collapsed 'Disagree' category. 


Discussion 

Most of the published descriptions and research on PRS involved large class sections in 
which the intent was to increase student engagement during lectures. Most professors 
would agree that engaging students is particularly difficult in these very large sections, but 
would perhaps question whether there is a need for PRS in classes that contain fewer 
students. Each of the physical therapy classes in which this investigation was conducted 
contained only 22 or fewer students, and these students agreed with the studies of students 
in large classes that PRS is useful. The physical therapy students particularly appreciated 
that PRS facilitated discussion and explanation, enabling them to achieve a deeper and 
stronger understanding of important material. These results also support the very limited 
results of other investigators in graduate education, which suggested that even graduate 
students in smaller classes felt that the technology was beneficial, especially because they 
could self-assess their understanding without the risk of being embarrassed in front of peers 
or instructors and that it helped them see which concepts an instructor thinks are 
important. The students in the current study felt that PRS questions also gave them an 
opportunity to practice applying concepts and to develop critical thinking and problem 
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solving skills. This may be related to the nature of a health professions curriculum in which 
clinical application of information is emphasized and in which critical thinking and problem 
solving skills are stressed, but this is an important finding. Teaching students how to apply 
learned information in novel situations is a critical component of physical therapy education. 
Each patient encountered in the clinical environment presents the student with a new 
combination of signs and symptoms that must be evaluated and integrated into a diagnosis, 
prognosis, and plan of care. It is important that students are given the opportunity to 
practice making sense of findings and rendering an appropriate judgment about action that 
is required. A few of the focus group comments suggested that the degree of application 
and problem solving that was required depended on the construction of PRS questions, 
which they reported differed between instructors. As with written examinations, instructors 
may approach quiz construction differently. Question type may also be expected to vary 
somewhat depending on the content being taught. Anatomy instruction, for example, 
typically involves more recall of names and locations of structures with little evaluation 
required, while patient examination necessitates continuous evaluation and synthesis of 
information as it is received. 

These student respondents observed that the type of questions used and the format of the 
quiz activities significantly affected how useful PRS was for helping them learn. Quiz 
questions were only helpful in preparing for exams, the students wrote, if they were similar 
to the type of questions that would be seen on the exams. They recognized that there were 
limitations to the multiple choice format and that not every exam question was expected to 
be multiple choice, but apparently also recognized a difference among various types of 
multiple choice questions. These graduate students were able to discern the difference and 
the relative usefulness of recall questions versus application questions, for example. Also, 
several focus group participants commented on the importance attached to the discussion of 
the questions and answers that followed each item on a quiz. Students felt that regardless 
of whether they answered a question correctly, the discussion helped them analyze and 
clarify the reasoning behind why each item was or was not the best choice, and recognized 
that this was the time in which a great deal of their learning occurred. 

Students largely preferred that PRS quizzes be ungraded. Focus group comments suggested 
that ungraded quizzes allowed students to focus on the discussions that followed questions 
with less anxiety about the total grade. They expressed that they learned a great deal from 
the quizzes even when their grades were very low, and by not counting the quiz grades, 
they could attend more to the learning. Because they felt that the quizzes provided good 
test preparation, these students also strongly preferred to have the quizzes provided to 
them after class so that they could use them to study. When the quizzes were not going to 
be provided to them, students busily tried to type questions, answers, and explanations 
during the quizzes; they felt that this also detracted from their ability to focus on the 
important discussions that occurred during quizzes. 

Students in this program were required to buy their response pads and then pay a fee to 
register them each semester. The largest drawback expressed by these students was 
related to this expense, but they qualified their response. They only felt that the expense 
was not justified if the professor did not use PRS regularly enough in class. Most of these 
students felt that using PRS quizzes once weekly justified the cost of the response pads. 

The other concern expressed by some students was the feeling of having to rush to answer 
PRS questions. Some said that they felt they did not have as much time as needed for 
thoughtful consideration of the answer choices because their classmates were waiting on 
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them, and they wanted to hurry so others would not have to wait. Some of these students 
said they preferred online quizzes to PRS quizzes for that specific reason. Students in other 
published investigations have not expressed this concern, but this apparent discrepancy 
may be related to research methodology more than to a true difference. Most published 
research used questionnaires that were constructed by faculty who were trying to answer 
specific concerns of the investigating faculty member, or that the faculty member assumed 
would be important to students. Investigators may not have recognized that time pressure 
would be of concern to some students, so it was not even offered as a survey item. In other 
words, students in previous investigations may in fact have also felt too rushed with in-class 
PRS quizzes but were never asked about this particular concern. 

The methodology used in the current study was unique and had several advantages in being 
able to virtually hear and explore the student voice. First, the focus group was conducted in 
a way that allowed anonymity between the facilitator and student, as well as between 
students. Interaction occurred online, and students signed into the focus group using a 
pseudonym. Students knew that the focus group facilitator was not one of the professors 
who used clickers and that their comments would only be communicated to other professors 
under their pseudonyms; there was no need to fear retribution related to negative 
comments. Secondly, with multiple students in each group, participants built on the 
comments made by other participants, supplementing, clarifying or extending the 
statements as needed. Because of the discussion-based format, student opinions were 
expressed in each student's own language. Finally, and most importantly, the entire group 
of current physical therapy students was given the opportunity to respond to the statements 
made by the sample of students in the focus groups. In this way, researchers could assess 
whether opinions of the focus group were representative of the whole group. This mixed 
method approach thus allowed exploration and interaction which provided the depth and 
texture of qualitative research, and validation by a larger sample which provided more 
quantitative insight into the larger group's opinions. This combined methodology effectively 
amplified the student voice. 

Previous investigations have identified technical problems as a frequent student concern. 

The students in this investigation agreed that technical problems sometimes wasted class 
time, but based on the proportion of students who agreed to this statement (60.5%), 
further examination was warranted. Some of the focus group comments suggested that at 
least some of the technical problems appeared to have been related to the technical 
expertise of the professor, in other words user error, rather than inherent system hardware 
or software problems. Previous investigations have also found that many students resent 
having PRS used to monitor attendance or feeling like they are being forced to attend class 
because PRS is being used for graded quizzes. Students in the current investigation did not 
mention these concerns, presumably since this is a small, graduate professional curriculum 
with consistent student attendance that is not affected by PRS use. 

Many physical therapy practitioners believe that the initial process of interviewing a patient 
in detail about the nature of his or her problem, what the patient believes caused the 
problem, and the effects of various positions and activities on that problem is the most 
important part of the first patient visit, even more important that the information obtained 
when the therapist performs physical tests and measurements. "If you ask the right 
questions and listen to the answers, the patient will tell you what is wrong," the saying 
goes. Perhaps the same is true of students. If we ask the right questions they will tell us 
what helps them learn. Without specific instruction or metacognitive activities, these 
graduate health professions student participants were able to differentiate between types of 
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questions, usefulness of the various types of questions for various purposes, and specific 
ways in which their learning was or was not augmented. They also recognized when 
technology was being used because the tool was available rather than as an intentional tool 
to enhance student understanding, as well as when user error interfered with the potential 
of the technology. They were even able to comment on and make specific recommendations 
related to the cost-benefit ratio of class time spent on quizzes versus the learning outcome 
achieved. College students have valuable experience and perspectives, and their voice can 
and should be incorporated into efforts to improve the quality of teaching and learning, 
including effective use of personal response systems. 


Recommendations for Practice 

The results of this inquiry have been very useful for understanding student perceptions of 
PRS, particularly related to aspects of use that they felt positively impacted their learning. 
These results echoed the positive student perceptions expressed in previous studies, but the 
mixed methodology added depth and variety to the types of concerns that have been 
addressed in the past. These results also emphasize the specific value that graduate 
physical therapy students perceive; this should stimulate other physical therapy faculty 
members, and faculty members in other graduate or health professions disciplines, to 
investigate the feasibility of use in their own curricula. 

PRS use is becoming more and more widespread in higher education, which is resulting in 
more and more students being required to invest in PRS response pads. An internet search 
in early 2012 revealed that many campuses have only recently begun adopting a campus 
standard that would dictate which specific response system should be used by all individual 
professors on that campus. Adoption of a campus standard has multiple benefits, including 
minimizing student expense. Without a campus standard, different professors adopt and 
require the use of different brands of PRS. This means that in many cases individual 
students must buy multiple brands of clickers since the response pads are not 
interchangeable from company to company. Adopting a campus standard also benefits 
faculty members since they can more easily share the benefits of experience with one 
specific system and grow the expertise of the community more efficiently. Technology 
support services could also be streamlined with a campus-wide PRS standard. 

PRS and other questioning methods foster learning by requiring that learners actively 
process information as they are attempting to solve the problem. This is termed generative 
learning (Mayer et al., 2009; Nelson, 2008). PRS also enhances the classroom experience 
through active learning, providing feedback, and increasing student attention; and all of 
these characteristics help increase student motivation (Nelson, 2008). Learning can be 
enhanced even further when PRS is used in conjunction with other pedagogical approaches. 
Corcose and Monty (2008) recommend "pairing clickers in a meaningful way with other 
pedagogical techniques such as peer discussion and peer teaching...as opposed to simply 
plugging them into existing course plans haphazardly" (p. 57). There are a number of 
descriptions in the higher education literature of ways that PRS can be used to support a 
variety of good pedagogical approaches to learning. Many professors have found that PRS 
facilitates other methods of obtaining student engagement and interaction. PRS also enables 
these professors to quickly and easily assess the effects of these approaches. 


Recommendations for Further Research 
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The use of PRS in higher education is increasing and spreading through multiple disciplines, 
but published investigations into effectiveness and student perceptions of PRS have largely 
focused on undergraduate students in large science and psychology classes. Very little 
research has been conducted in graduate education. Very little research is available in the 
health professions, and what has been published is nearly exclusively in nursing. This 
investigation represents an attempt to address the lack of published research in both of 
these important areas. 

Few studies in the published PRS literature have examined student factors that might be 
related to preferences or effectiveness. One may wonder, for example, whether students of 
color have different preferences and opinions than white students, or whether non-native 
English speakers have different preferences than native English-speakers. More studies are 
needed to see whether meaningful differences exist between male and female preferences 
related to PRS. Nothing in our study suggests that this is the case, but our population was 
82% female, 95% white, and 100% native English speakers, which was 


https://doi.org/l0.20429/ijsotl.2012.060226 


13 



Student Opinions and Preferences Regarding Personal Response 


too homogenous for meaningful analysis. In larger, more diverse populations, this 
approach would perhaps be useful for attempting to add more depth to our understanding 
of student preferences and to be sure that learning through the use of this technology is 
being optimized for all students. 

The methodology used in this study is one that combines the considerable advantages of 
qualitative inquiry with concise representations of trends provided through descriptive 
statistics, and could be used in a wide variety of scholarship of teaching and learning 
projects. By utilizing online technologies, additional advantages were realized. Focus group 
participants were able to participate anonymously, which may have increased their 
willingness and honesty in responding to the facilitator and in expressing negative opinions 
or disagreement. Neither focus group participants nor survey respondents had to be in 
physical proximity. In fact, some of the focus group participants in this study were residing 
temporarily in other states for clinical training. The online chat feature of the course 
management software maintains a written record of all communication during a focus group 
conducted in a chat room, and so the tedious and potentially inaccurate transcription of 
audio recordings was avoided. Also, the printed transcript made it easy to link the 
participants with their comments; this is sometimes difficult with audio recordings of group 
discussion. 

The online focus group makes it impossible for the facilitator or other participants to read 
the body language and voice quality characteristics that allow for additional interpretation of 
subtle shades of meaning that are often easier to recognize in a face-to-face focus group. 
Also, participation in the online conversation may be impeded by the inability to read or 
type quickly; it can be difficult to follow a conversation in a synchronous chat since, in 
essence, the technology allows many people to 'speak' at the same time. For this same 
reason, it may be easier for a dominant participant to be over-represented in the online 
environment as there is no need to wait while others are speaking, though the reverse is 
also true: students who may be hesitant to speak in person may feel more comfortable 
participating in a written conversation. 

Today's students expect to be engaged and to use technology to enhance their learning 
(Lindbeck & Fodrey, 2010). PRS is one option for increasing student learning, engagement, 
and attention, and can be a useful way of promoting application of knowledge in various 
disciplines, as well as problem solving and critical thinking. The results of this study further 
inform faculty members' understanding of student opinions, perceptions, and suggestions 
related to effective PRS use, which provides another source of information that can be used 
for revising or adopting sound pedagogical approaches to the use of PRS in the higher 
education classroom. 
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Appendix. 

Guiding questions for student focus groups 

Thank you for participating in this project. We are genuinely interested in your thoughts 
regarding the "clickers". Your responses are anonymous - we are interested in both positive 
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and negative opinions. Please feel free to give us your honest and thoughtful feedback in 
order to better understand the 'student-side' of clicker use. 


Their history with clickers before PT school 

1. Good use of class time? 

2. Biggest complaint about them? 

3. Biggest advantage of them? 

Their use in the PT program 

4. Good use of class time? 

5. How do they relate to your performance on or preparation for other activities, such 
as exams? 

6. Compare clickers to Vista quizzes (perhaps including whether their preparation is any 
different) 

7. Would you like to see more (or less) use of clicker quizzes in classes (this could 
mean more frequent use in any one class or could mean used in additional classes) 

8. Biggest complaints or disadvantages of using them 

9. Biggest advantage of using clickers 

10. Do clickers benefit some types of students more than others? 

11. **Do some types of questions work better than others (and better in what wav? ) 

12. How does the nature of clicker quizzes change when the grades are 'counted' or not? 

13. Do you know of other ways that clickers have been used in other classes before PT 
school that would be useful in your PT classes? Please describe 

Other comments? 

Thank you again for your time 
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